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1.  Introduction 


Classification  using  the  MetalMapper  advanced 
electromagnetic  sensor  was  demonstrated  at  the  former 
Spencer  Artillery  Range,  TN  in  2012.  Three  parallel 
demonstrations  were  conducted  at  this  site.  This  report 
summarizes  the  results  of  the  MetalMapper  demonstration 
in  the  “Open  Area.”  Later  reports  will  document  the 
results  of  the  demonstrations  in  the  “Dynamic”  and 
“Wooded”  areas.  The  document  Implementing  Classification 
on  Munitions  Response  Sites  (Ref.  1)  provides  practical 
information  for  deciding  whether  classification  is 
appropriate  to  a  particular  site  and  how  it  is  best 
implemented. 

Classification  is  motivated  by  the  need  to  perform 
munitions  response  more  cost-effectively  so  that  limited 
clean  up  dollars  can  be  used  to  reduce  real  risk  on 
munitions-contaminated  sites  sooner.  The  estimated 
liability  in  the  FY 1 0  Defense  Environmental  Programs 
Report  to  Congress  for  Munitions  Response  is  $15. 2B. 

(Ref.  2)  The  bulk  of  this  liability  is  $10. OB  for  the  1703 
sites  identified  in  the  Formerly  Used  Defense  Sites 
(FUDS)  program  and  $4.4B  for  the  2433  sites  identified 
on  Active  Installations.  The  remaining  $0.8B  is  in  Base 
Realignment  and  Closure  (BRAC).  The  estimated 
completion  dates  for  many  sites,  particularly  in  the  FUDS 
program,  are  decades  out  if  they  are  to  be  cleaned  up  at 
planned  funding  levels  using  current  practice. 

When  a  munitions  response  site  is  cleaned  up,  in  most  cases,  it  is  mapped  with  a  geophysical  sensor 
and  the  locations  of  all  detectable  signals  are  excavated.  Geophysical  sensors  detect  metal  and, 
therefore,  many  of  the  detections  do  not  correspond  to  munitions,  but  rather  to  harmless  metallic 
objects.  Field  experience  indicates  that  95-99%  or  more  of  objects  are  found  to  be  nonhazardous. 
Currently-used  technology  does  not  provide  a  means  to  discriminate  between  munitions  and  other 
items,  termed  “clutter.”  As  a  result,  most  of  the  costs  to  remediate  a  munitions-contaminated  site 
using  current  methods  are  spent  on  excavating  targets  that  pose  no  threat. 

Classification  is  a  process  used  to  make  a  decision  about  the  likely  origin  of  a  signal.  In  the  case  of 
munitions  response,  high-quality  geophysical  data  can  be  interpreted  with  physics-based  models  to 
estimate  parameters  that  are  related  to  the  physical  attributes  of  the  object  that  resulted  in  the  signal, 
such  as  its  physical  size  and  aspect  ratio.  The  values  of  these  parameters  may  then  be  used  to 


Spencer  Artillery  Range,  TN,  Open 
Area  -a  variety  of  projectiles  expected, 
flat  terrain  with  open  access,  minimal 
geologic  inte  ference,  medium  anomaly 
density 

Munitions  —  37 -mm,  75-mm,  76-mm, 
105-mm  and  155-mm  projectiles 

Results  —Production  contractor  field 
crews  from  two  different  vendors  collected 
cued  MetalMapper  data.  Both 
production  contractor  geophysicists  and 
the  developers  of  classification  methods 
were  successful  in  using  these  data  to 
achieve  substantial  classification.  Most 
analysts  correctly  identified  all,  or  all  but 
one,  TOI  while  correctly  classifying  75 
to  90%  of  the  clutter.  The  best  predictor 
of  classification  peformance  was  analyst 
experience  which  emphasises  the 
importance  of  continued  training. 
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determine  whether  the  signal  arose  from  a  munition  or  harmless  clutter.  With  reliable  classification, 
only  the  munitions  need  to  be  removed  from  the  site. 

The  Strategic  Environmental  Research  and  Development  Program  (SERDP)  and  the  Environmental 
Security  Technology  Certification  Program  (ESTCP)  have  supported  the  development  of  purpose- 
built  advanced  electromagnetic  sensors  and  associated  analysis  methods  for  classification.  Following 
the  successful  demonstration  of  classification  methods  in  controlled  test  environments,  ESTCP 
initiated  a  Classification  Pilot  Program  to  validate  the  application  in  real-world  conditions.  The  goal 
of  the  program  is  to  demonstrate  that  classification  decisions  can  be  made  using  an  explicit 
approach,  based  on  principled  analysis  that  is  transparent  and  reproducible.  The  demonstrations  are 
planned  and  conducted  in  cooperation  with  regulators  and  program  managers  in  the  Services. 

The  physics  governing  the  electromagnetic  response  of  a  metal  object  is  well  understood  and 
predictable.  Data  collected  with  these  sensors  contain  the  same  information  content  on  any  site  and 
demonstrations  to  date  have  confirmed  that  classification  works  predictably.  Nevertheless, 
demonstrations  will  be  required  at  a  number  of  sites  to  represent  the  wide  variability  in  munitions 
types,  target  densities,  terrain,  vegetation,  geology,  land  use  history,  future  land  use,  and  other  site 
characteristics  that  will  affect  the  applicability  of  classification  and  to  establish  cost  effectiveness  and 
implementability.  The  demonstrations  also  present  an  opportunity  to  work  out  standard  operating 
procedures  and  establish  quality  control  (QC)  measures.  Prior  demonstrations  have  been  conducted 
at  a  number  of  sites  across  the  country;  details  about  past  and  ongoing  demonstrations  can  be  found 
on  the  SERDP-ESTCP  web  site  at  http:/ / serdp-estcp.org/Featured-Initiatives/Munitions- 
Response-Initiatives  /  Classification-Applied-to-Munitions-Response. 

The  demonstration  at  Spencer  Artillery  Range  continues  the  practice  of  data  collection  by 
production  contractors  and  data  analysis  by  both  production  geophysicists  and  the  developers  of  the 
analysis  methods.  This  is  an  important  consideration  in  evaluating  and  applying  the  results;  the 
production  geophysicists  are  far  less  experienced  than  the  developers  and  any  performance 
shortcomings  should  be  attributed  to  inexperience.  Table  1  shows  the  participants  and  their  roles  in 
the  Spencer  demonstration. 


Table  1.  Participants  in  the  MetalMapper  Demonstration  at  former  S 

rencer  Artillery  Range 

Task 

Performer(s) 

Task 

Performer(s) 

Site  Preparation 

URS,  Corp. 

Data  Analysis 

Dartmouth  College 

NAEVA 

SAIC 

Shaw 

SIG,  Inc. 

Sky  Research,  Vancouver* 
URS,  Corp 

USACE 

MetalMapper 

Data  Collection 

URS,  Corp 

NAEVA 

Intmsive 

Investigation 

URS  Corp 

Scoring 

Institute  for  Defense 
Analyses 
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2.  Spencer  Demonstration  Flow 

The  sequence  of  the  demonstration  is  outlined  in  the  flow  chart  in  Figure  2-1. 


Figure  2-1.  Flow  chart  outlining  steps  in  the  demonstration  at  Spencer  Artillery  Range.  Blue  boxes 
are  tasks  performed  by  ESTCP.  Others  are  tasks  performed  by  contractors. 

Prior  to  the  beginning  of  data  collection,  an  instrument 
verification  strip  (TVS)  was  installed  and  the  site  was  seeded 
with  inert  munitions  and  small  (1-in  nominal  x  4-in  Schedule 
80  pipe  nipples)  and  medium  (2-in  nominal  x  8-in  Schedule 
40  pipe  nipples)  industry  standard  objects  (ISOs).  (Ref.  3) 

The  data  collection  teams  visited  the  IVS  twice  daily  to 
verify  equipment  function  at  the  start  and  end  of  each  day. 

Since  there  are  few  native  unexploded  ordnance  (UXO)  on 
any  munitions  response  site,  the  seeds  provided  sufficient 
targets  of  interest  (TOI)  to  allow  a  statistically  defensible 
determination  of  the  correct  classification  of  TOI. 

The  site  was  surveyed  with  an  EM61  to  provide  an  initial  list 


Targets  of  Interest  (TOI)  are  all  objects 
that  must  be  removed from  the  site. 
Typically  the  TOI  will  include  all 
known  or  suspected  munitions  types,  any 
other  unexpected  munitions,  munitions 
parts  such  as  fuyes  that  present  an 
explosive  hazard,  and  all  seeded  items. 
When  classification  is  applied  to  a  site, 
the  local project  team  will  decide  what 
items  constitute  TOI. 
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of  detected  anomalies.  The  MetalMapper  was  used  to  collect  cued  data  over  each  anomaly.  All 
detected  targets  were  dug  up  to  provide  complete  ground  truth  for  the  purposes  of  determining 
performance.  The  UXO  technicians  photographed  each  item  that  was  dug  and  recorded  its 
location,  depth,  and  description. 


The  cued  geophysical  data  were  passed  to  the  data  analysis  teams.  A  complete  overview  of  the 
analysis  procedures  can  be  found  in  Ref  1 .  Briefly,  the  analysts  used  methods  based  on  the  dipole 
model  to  estimate  target  parameters.  Analysts  were  offered  training  data  from  test  pit  measurements 
and  the  opportunity  to  request  additional  training  data  from  the  recovered  targets,  as  though  they 
were  doing  a  limited  number  of  sample  digs.  These  data  were  used  to  set  classifier  rules  -  the 
decisions  that  separate  the  anomalies  into  TOI  and  non-TOI.  The  classifiers  were  then  applied  to 
all  of  the  targets  that  remained  blind  for  each  demonstrator.  Since  training  data  was  by  request,  the 
blind  target  set  was  different  for  each  demonstration. 


The  product  required  from  each  analyst  was  a  ranked  anomaly  list;  an  example  is  shown  in  Figure 
2-2.  One  and  only  one  judgment  is  required  for  each  entry  on  the  anomaly  list.  The  first  items 
below  the  training  data  on  each  anomaly  list  are  those  targets  for  which  reliable  parameters  cannot 
be  extracted  and  therefore  must  be  dug.  Next  are  those  items  which  the  analyst  is  the  most 
confident  are  TOI.  These  items  are  ranked  according  to  decreasing  likelihood  that  the  item  is  a 
TOI.  Any  items  which  the  analyst  was  able  to  analyze  but  was  not  able  to  make  a  classification 
decision  on  at  this  time  were  placed  next  on  the  anomaly  list.  Last  are  all  those  items  that  the  analyst 
was  confident  are  not  TOI  ranked  by  their  likelihood.  The  analysts  were  required  to  specify  a  size 
estimate  for  any  item  marked  to  be  dug.  This  initial  list  is  shown  in  the  left  panel  of  Figure  2-2. 


Initial  Ranked  Anomaly  List 


Anomaly 

ID 

Dig  on 
First  Pass 

Type 

Comment 

718 

Y 

Classifier  training  data 

207 

Y 

2498 

Y 

Unable  to  extract  reliable  parameters 

811 

Y 

81  mm 

313 

N 

Unable  to  classify 

883 

N 

N 

N 

N 

High  likelihood  not  TOI 

N 

N 

N 

N 

N 

N 

Final  Ranked 
Anomaly  List 


Anomaly 

ID 

Dig 

Type 

718 

Y 

207 

Y 

2498 

Y 

883 

N 

N 

N 

N 

N 

N 

N 

N 

N 

N 

Figure  2-2.  Example  initial  and  final  Ranked  Anomaly  Lists.  A  detailed  description  is  in  the  text. 
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The  seeds  were  divided  into  QC  seeds  and  blind  seeds.  When  analysts  submitted  their  initial 
prioritized  lists,  the  QC  seeds  were  used  to  provide  feedback  if  seed  targets  were  missed.  Analysts 
were  also  provided  with  the  ground  truth  information  on  all  anomalies  in  the  red  part  of  their  lists 
and  any  requested  anomalies  in  the  yellow  part.  This  is  signified  by  the  threshold  on  the  left  list  in 
Figure  2-2.  Based  on  this  information,  the  analysts  were  then  allowed  to  revisit  their  rankings  and 
assignments  for  all  items  that  were  still  blind  until  they  were  satisfied  that  the  best  possible 
classification  had  been  achieved. 

In  the  final  list,  shown  in  the  right  panel  of  Figure  2-2,  the  analyst  is  required  to  provide  a  threshold 
that  corresponds  to  the  division  between  those  items  recommend  for  digging  and  those  that  can 
safely  remain  in  the  ground.  That  is,  every  item  on  the  list  is  marked  to  be  dug  or  not  with  a 
threshold  separating  the  two  categories.  The  final  prioritized  anomaly  lists  submitted  in  this 
demonstration  were  scored  against  the  emplaced  blind  seeds  and  recovered  targets  by  IDA. 

3.  Site  Description  and  Preparation 

The  former  Spencer  Artillery  Range  is  a  30,618  acre  site  located  near  Spencer,  Tennessee.  (Ref.  5) 
Construction  of  the  range  began  in  1941  and  documentation  identifies  establishment  of  two  impact 
areas:  Jakes  Mountain  (5,060  acres)  and  Bald  Knob  (2,090  acres).  Troop  training  took  place  until 
September  1 944,  by  which  time  Army  ground  forces  had  either  departed  or  were  under  orders  to 
depart.  Subsequent  arrangements  were  made  for  Dyersburg  Army  Air  Field  to  use  the  Spencer 
Artillery  Range  as  an  air-to-ground  gunnery  range.  The  land  reverted  back  to  the  original  25 
leaseholders  in  the  summer  of  1946.  Several  surface  decontamination  sweeps  were  completed  on 
portions  of  the  former  range  in  the  1950s.  Since  then,  numerous  tracts  of  land  have  been  sold 
and/ or  subdivided,  significantly  increasing  the  number  of  property  owners  from  the  original  25  to 
several  hundred  landowners  today. 

The  demonstration  was  conducted  in  a  portion  of  Munitions  Response  Site  (MRS)  1.  An  aerial 
photo  of  the  site  is  shown  in  Figure  3-1  with  the  three  parallel  demonstration  areas  marked. 

All  visible  metal  objects  were  removed  from  the  surface  at  the  site.  First  order  reference  points  were 
installed  by  a  registered  surveyor  for  geolocation  reference.  An  area  free  of  contamination  was 
located  near  the  demonstration  area  to  establish  an  IVS  used  for  daily  verification  of  proper  sensor 
operation  and  a  training  pit  to  collect  sensor  data  for  algorithm  training. 

The  known  munitions  at  this  site  include  37-mm,  75-mm,  76-mm,  105-mm,  and  155-mm  projectiles. 
The  analysts  were  provided  information  about  the  historical  use  and  known  munitions.  In  addition 
to  these  munitions,  demonstrators  were  directed  that  any  unexpected  munitions  would  also  be 
considered  TOI. 

The  final  demonstration  site  was  seeded  with  enough  TOI  to  ensure  statistical  validity  on  measures 
of  classification  of  targets  of  interest.  The  TOI  seeds  are  listed  in  Table  3-1.  The  seeds  included 
not  only  inert  projectiles,  but  also  small  and  medium  industry  standard  objects  (ISOs).  (Ref.  3)  The 
ISOs  are  also  considered  TOI  and  expected  to  be  both  detected  and  correctly  classified. 
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Figure  3-1.  Aerial  photo  of  the  former  Spencer  Artillery  Range  showing  the  demonstration  areas. 
The  Open  Area  is  outlined  in  blue,  the  Dynamic  Area  in  green  and  the  Wooded  Area  in  orange.  The 
Open  and  Dynamic  Areas  have  been  logged  since  the  time  of  this  photograph. 

For  safety,  seeds  were  emplaced  using  standard  anomaly  avoidance  procedures.  No  attempt  was 
made  to  separate  the  seeds  from  the  surrounding  clutter.  For  realism,  the  emplacement  teams  were 
instructed  to  replace  any  metal  dug  up  during  emplacement  back  in  the  hole  with  the  seeded  object. 
To  further  test  the  analysts’  ability  to  distinguish  multiple  objects  in  the  field-of-view  of  the  sensor, 
18  of  the  seeds  were  emplaced  with  one  or  two  fragments  either  20  or  30  cm  away. 


Table  3-1.  Seeds  Emplaced  in  the  Open  Area  at  Spencer  Range 


Item 

Number 

Depth  Range  (cm)* 

Small  ISO80 

28 

15-30 

37-mm  projectile 

28 

15-30 

Medium  ISO40 

5 

15-45 

60-mm  mortar 

5 

15-45 

75-mm  projectile 

14 

15-46 

105-mm  projectile 

1 

45 

155-mm  projectile 

1 

45 

^Depths  are  to  the  center  of  the  object. 
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4.  EM61  Detection  Survey 

An  initial  survey  was  performed  with  an  EM61-MK2  in  its  standard  cart  configuration  with  cm-level 
global-positioning-system  (GPS)  navigation.  These  data  were  used  to  provide  a  common  anomaly 
list  for  the  two  MetalMapper  data  collections  that  were  to  follow.  No  classification  was  attempted 
using  these  data  as  it  has  been  shown  previously  that  these  data  are  not  useful  for  classification. 

(Ref.  6) 

The  data  quality  objectives  for  the  detection  survey  were  based  on  the  37-mm  projectile,  which  was 
expected  to  be  the  most  difficult  to  detect  TOI  at  the  site.  The  EM61  survey  was  performed  on 
half-meter  line  spacing.  The  anomaly  selection  criteria  were  set  to  detect  a  37-mm  projectile  at  a 
depth  of  34  cm.  This  depth  was  chosen  as  the  deepest  depth  to  which  a  37-mm  could  be  reliably 
detected.  The  EM61  signal  strength  in  channel  2  versus  depth  for  the  37-mm  is  shown  in  Figure  4- 
1.  (Ref.  7)  The  signal  at  34-cm  depth  for  the  least  favorable  orientation  is  4.0  mV,  which  was  used 
as  the  amplitude  threshold  for  identifying  anomalies. 


Depth  (cm  bgs)  Assuming  Standard  Wheels 
0  20  40  60  80 


Figure  4-1.  EM61  response  in  Gate  2  versus  depth  for  a  37-mm  projectile 

The  EM61  survey  resulted  in  a  total  of  1104  anomalies  in  the  4.4-acre  Open  Area,  including  the 
seeds.  This  translates  to  approximately  250  anomalies  per  acre.  Details  can  be  found  in  the 
vendor’s  report.  (Ref.  7)  All  seeds  were  detected  in  the  EM61  survey. 

5.  MetalMapper  and  Data  Collection 

The  MetalMapper  developed  by  Geometries  is  designed  to  be  a  stand-alone  survey  and  cued 
detection  system.  The  system,  shown  in  Figure  5-1,  is  composed  of  three  orthogonal  1-m  x  1-m 
transmitters  for  target  illumination  and  7  three-axis  receivers  for  recording  the  response.  Its 
sampling  is  electronically  programmable  and  therefore  flexible.  It  measured  the  decay  curve  up  to  8 
ms  after  the  transmitters  were  turned  off.  Centimeter-level  GPS  is  used  for  navigation  and 
geolocation  and  an  inertial  measurement  unit  (IMU)  is  used  to  measure  platform  orientation.  In 
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cued  mode,  MetalMapper  is  positioned  over  each  anomaly  on  its  target  list  and  collects  the  full  suite 
of  data  while  stationary.  The  digital  data  set  produced  by  MetalMapper  is  fully  described  in  Ref.  4. 


Figure  5-1.  Schematic  and  photo  of  the  MetalMapper  as  used  at  former  Spencer  Range 

In  this  portion  of  the  demonstration  MetalMapper  was  used  only  in  cued  mode.  It  was  deployed  in 
a  sled  configuration  mounted  to  the  front  of  a  tractor.  Two  commercial  geophysics  vendors,  URS, 
Inc.  and  NAEVA,  collected  MetalMapper  data  at  Spencer  Range.  Operators  were  trained  in  the 
field  operations  and  QC  procedures  and  data  were  initially  QCed  by  personnel  with  prior  experience 
with  MetalMapper. 

Details  on  the  data  collection  and  QC  procedures  followed  by  each  vendor  can  be  found  in  the 
respective  reports.  (Ref.  8,  9)  The  most  common  QC  failure  was  that  the  MetalMapper  was 
positioned  too  far  from  the  anomaly  to  obtain  reliable  parameter  estimates.  If  the  separation 
between  the  center  of  the  MetalMapper  and  the  anomaly  location  was  more  than  40  cm,  the 
anomaly  was  revisited  and  additional  data  collected  within  the  40-cm  specification. 

Both  data  sets  were  analyzed  by  multiple  analysts.  Although  there  were  small  differences  in  various 
measures  of  data  quality,  both  data  sets  were  of  high  overall  quality  and  acceptable  to  perform 
classification.  Detailed  comparisons  of  the  analyses  of  the  two  data  sets  can  be  found  in  the 
individual  analysts’  reports  available  on  the  SERDP-ESTCP  web  site  (http:/ / serdp- 
estcp.org/Featured-Initiatives/Munitions-Response-Initiatives/ Classification- Applied-to-Munitions- 

Response/  Former-Spencer- Artillery-Range). 

The  production  rates  for  the  two  data  collectors  were  214  and  228  targets  per  day  for  URS  and 
NAEVA  respectively.  URS  reported  recollecting  data  on  about  5%  of  the  total  anomalies  due  to 
QC  failures,  while  NAEVA  reported  an  overall  rate  of  14%.  Both  contractors  noted  that  these  rates 
reflect  inefficiency  inherent  in  learning  to  operate  a  new  system  and  are  not  likely  predictive  of 
future  productivity. 


6.  Intrusive  Investigation 

A  summary  of  the  results  of  the  intmsive  investigation  is  shown  in  Figure  6-1.  The  left  panel  in  the 
figure  shows  the  distribution  of  the  number  of  items  that  were  recovered  from  a  single  hole.  As 
expected  given  the  moderate  anomaly  density  at  this  site  (~250/ acre),  more  than  85%  of  the 
anomalies  were  caused  by  a  single  buried  item.  The  pie  chart  on  the  right  hand  side  of  Figure  6-1 
shows  the  identities  of  the  recovered  objects.  The  majority  of  the  recoveries  (~90%)  were  identified 
as  munitions  debris;  not  unexpected  for  an  impact  area.  Only  ten  items  were  identified  as  “other 
debris.” 


Number  of  Items  Recovered  per  Anomaly 


Identity  of  Recovered  Items 


Figure  6-1.  Distribution  of  items  recovered  per  anomaly  (left  panel)  and  identity  of  recovered  items 

(right  panel) 

Three  non-seed  TOI  were  recovered  as  part  of  this  demonstration;  none  of  the  three  required 
detonation  by  the  UXO  crew. 

The  depth  distribution  of  the  recovered  items  is  given  by  the  blue  line  in  Figure  6-2.  As  at  most 
munitions  sites,  the  vast  majority  of  items  were  recovered  in  the  top  10  to  20  cm.  The  depths  of  the 
three  recovered  TOI  are  indicated  by  the  stars  and  the  depth  range  of  the  seeds  by  the  grey  bars. 

As  has  been  the  case  in  all  the  ESTCP  demonstrations  to  date,  the  seed  items  were  buried  deeper 
than  the  vast  majority  of  items  recovered  from  the  site.  One  75-mm  projectile,  however,  was 
recovered  at  64  cm  which  is  below  the  depth  range  of  the  75-mm  seeds.  Other  than  this  item,  the 
seeds  proved  to  be  a  conservative  QC  test. 
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Number  of  Recovered  Items 


0  50  100  150  200  250  300 


Seed  Depth  Range 


Figure  6-2.  Depth  distribution  of  recovered  items  from  Spencer  Range  (blue  line).  The  depths  of 
the  three  TOI  recovered  in  the  Open  Area  are  marked  by  stars  and  the  depth  range  of  the  QC  seeds 

are  shown  as  grey  bars. 


7.  Data  Analysis 

All  data  sets  were  analyzed  by  multiple  analysts,  including  both  the  developers  of  the  analysis 
methods  and  production  geophysics  vendors.  Figure  7-1  shows  an  overview  of  the  results  achieved 
by  all  analysts  working  with  the  MetalMapper  data.  The  panel  on  the  left  shows  the  percent  of  TOI 
correcdy  classified  versus  the  number  of  clutter  at  each  analyst’s  operating  threshold.  The  inset 
expands  the  upper  left  of  the  graph  for  clarity.  Desired  performance  is  to  correctly  classify  100%  of 
the  TOI  and  eliminate  all  of  the  clutter.  The  panel  on  the  right  shows  the  number  of  clutter  that  the 
analyst  needed  to  dig  to  get  to  100%  correct  classification  of  the  TOI,  regardless  of  where  the 
analyst  put  the  threshold.  This  can  be  thought  of  as  the  best  the  analyst  could  have  done  by  putting 
the  threshold  in  exactly  the  optimum  place  where  the  last  TOI  is  found  in  the  ranked  list.  The 
symbols  above  the  bars  correspond  to  the  symbols  in  the  left  panel. 

At  their  specified  thresholds,  most  analysts  achieved  good  results,  correctly  classifying  greater  than 
95%  of  the  TOI  and  eliminating  70-90%  of  the  clutter.  This  includes  both  the  classification 
algorithm  developers  and  the  production  contractors  and  is  quite  remarkable,  especially  considering 
that  this  was  the  first  attempt  at  analyzing  MetalMapper  data  by  many  of  these  analysts. 

This  range  is  even  more  evident  in  the  right  panel.  The  best  performer  could  have  eliminated  all  but 
52  of  the  clutter  (a  95%  reduction)  with  100%  correct  classification  of  TOI,  where  as  the  poorest 
performer  at  best  could  have  eliminated  about  only  18%  of  the  clutter. 
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Figure  7-1.  Results  from  all  analysts  in  the  Spencer  Open  Area  demonstration.  The  left  panel  shows 
the  percent  TOI  correctly  identified  versus  number  of  clutter  at  the  analysts  chosen  threshold.  The 
right  panel  shows  the  number  of  clutter  that  would  have  to  be  dug  to  classify  100%  of  the  TOI, 
regardless  of  where  the  analyst  put  the  threshold. 

7.1  Example  Analysis  Results 

An  analyst  from  Dartmouth  College  processed  the  cued  MPV  data  using  methods  developed  in 
house.  (Ref.  0)  The  results  of  this  analysis  are  shown  in  Figure  7-2.  The  colors  on  the  plot 
correspond  to  the  red  and  green  colors  in  the  final  ranked  anomaly  list  as  shown  in  Figure  2-2.  The 
red  are  the  items  the  analyst  classified  as  “high  likelihood  TOI”  and  the  green  are  those  the  analyst 
called  “high  likelihood  not  TOI.”  No  anomalies  were  classified  as  “unable  to  classify.” 

The  graph  plots  the  percent  of  the  TOI  correctly  classified  on  the  vertical  axis  and  the  number  of 
clutter  items  on  the  horizontal  axis.  The  offset  from  zero  in  the  starting  point  reflects  the  training 
data  that  the  analyst  requested.  Any  anomalies  classified  as  “can’t  extract  reliable  parameters”  would 
be  represented  by  an  initial  black  line;  no  anomalies  were  in  this  category  for  this  analysis.  The  blue 
dot  represents  the  threshold  selected  by  the  analyst  and  the  orange  dot  shows  the  point  on  the 
ranked  anomaly  list  where  100%  of  the  TOI  are  captured.  Ideally,  a  classifier  would  correctly 
identify  all  targets  of  interest  in  the  red  with  zero  clutter  and  all  of  the  clutter  would  be  in  the  green. 
In  this  case,  the  red  part  of  the  curve  would  go  straight  up  to  100%  and  the  green  part  of  the  curve 
would  run  straight  across  the  top  axis.  Success  in  these  demonstrations  was  defined  by  eliminating 
the  maximum  amount  of  clutter  while  correctly  identifying  all  of  the  TOI. 
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Figure  7-2.  Results  of  a  Dartmouth  analysis  of  the  MetalMapper  data  collected  by  NAEVA 

In  this  demonstration,  there  were  983  total  clutter  items  as  determined  from  the  ground  truth.  The 
Dartmouth  analyst  was  able  to  correctly  identify  all  but  85  of  these  items  at  her  threshold,  for  a 
possible  savings  of  more  than  90%  of  the  digs. 

An  analyst  from  the  US  Army  Corps  of  Engineers  also  achieved  good  results  using  the  NAEVA 
data,  Figure  7-3.  This  analyst  used  very  little  site-specific  training  data  (the  ROC  curve  starts  very 
near  the  origin)  and  was  able  to  identify  ~80%  of  the  TOI  very  efficiently  and  identify  100%  of  the 
TOI  while  saving  nearly  80%  of  the  clutter  digs.  The  difference  between  this  analyst  and  the 
Dartmouth  analyst  is  primarily  experience  with  these  techniques. 
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Figure  7-3.  Results  of  the  USACE  analysis  of  the  MetalMapper  data  collected  by  NAEVA 
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Similar  results  were  obtained  by  a  novice  analyst  from  Shaw  using  the  MetalMapper  data  collected 
by  URS  as  seen  in  Figure  7-4.  This  analyst  identified  70%  of  the  TOI  with  high  efficiency  and  100% 
of  the  TOI  while  saving  over  80%  of  the  clutter  digs. 
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Figure  7-4.  Results  of  the  Shaw  analysis  of  the  MetalMapper  data  collected  by  URS 

Not  all  analysts  correctly  identified  100%  of  the  TOI  at  their  threshold.  The  curve  in  Figure  7-5  is 
representative  of  a  number  of  analyses.  This  analyst  was  very  efficient  at  identifying  all  but  one 
TOI.  The  missed  TOI  was  a  small  ISO80  with  a  piece  of  frag  almost  directly  above  the  seed.  This 
is  a  case  where  an  analyst  with  greater  experience  with  the  multi-source  solvers  that  are  beginning  to 
become  standard  in  the  commercially-available  analysis  programs  may  have  identified  this  seed  item. 
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Figure  7-5.  Results  of  the  NAEVA  analysis  of  the  MetalMapper  data  collected  by  NAEVA 
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8.  Cost  Comparison 

The  demonstration  took  place  on  a  small  part  of  the  former  Spencer  Artillery  Range  and  incurred 
costs  for  many  items  specific  to  a  demonstration  that  would  not  be  needed  in  an  application  of 
classification  to  a  real  site.  Nevertheless,  we  can  extract  meaningful  projected  performance  for  the 
technology  and  apply  reasonable  industry  unit  costs  for  various  elements  to  arrive  at  a  total  cost 
comparison  for  clearing  an  example  100-acre  site  with  and  without  the  use  of  classification. 

We  made  the  following  assumptions: 

•  The  example  takes  place  in  an  area  with  similar  munitions  types  and  the  same  density  of 
anomalies  as  seen  in  this  demonstration.  There  were  approximately  250  anomalies  per  acre 
at  the  demonstration  site.  Extrapolating,  we  would  expect  about  25,000  anomalies  in  a 
similar  1 00-acre  area,  with  24,925  clutter. 

•  Mob/ demob  and  a  surface  sweep  would  be  required  for  either  scenario. 

•  Three  TOI  were  found  during  the  intmsive  investigation  of  the  ~ 4-acre  area.  In  a  100-acre 
site  we  predict  about  75  native  TOI. 

•  The  baseline  is  an  EM61  survey  with  0.5-m  line  spacing.  This  would  be  used  to  select 
anomalies  for  digging  without  classification  and  for  cueing  with  the  advanced  sensor  in 
hybrid  mode. 

•  Survey  data  will  be  collected  using  MetalMapper  in  dynamic  mode  for  the  advanced  sensor 
only  mode. 

•  The  site  is  seeded  at  a  rate  so  on  average  one  seed  will  be  encountered  each  day  of 
MetalMapper  data  collection.  With  an  estimate  of  25,000  total  anomalies  and  a  production 
rate  of  200  anomalies  per  day,  we  seed  125  inert  items.  These  QC  seeds  would  be  used 
whether  classification  was  used  on  the  site  or  not. 

•  The  classification  performance  is  as  achieved  by  the  production  firms  with  ~80%  of  the 
clutter  correctly  identified  and  remaining  undug. 

•  The  unit  costs  are  as  shown  in  Table  8-1. 


Table  8-1.  Unit  cost  assumptions 


Item 

Units 

Cost 

Mob/Demob 

1 

$7,000  j 

Surface  Sweep 

acre 

$2,500 

IVS  and  Seed  Emplacement 

125  seeds 

$22,650 

EM61  Survey  Data  Collection  and  Analysis 

acre 

$1,000  j 

Dynamic  MetalMapper  Data  Collection  and  Analysis 

acre 

$5,000 

Cued  MetalMapper  Data  Collection  and  Analysis 

per  anomaly 

$30 

Digs 

per  dig 

$125 

With  these  assumptions  the  costs  were  calculated  using  the  elements  shown  in  Table  8-2.  Using  a 
hybrid  approach  in  which  anomalies  are  identified  from  an  EM61-MK2  survey  and  cued  data  is 
collected  with  the  MetalMapper  (the  method  used  in  this  demonstration),  a  49%  savings  on  project 
field  work  can  be  realized. 
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As  the  Classification  demonstration  program  has  progressed,  it  has  become  apparent  that  a 
significant  number  of  anomalies  can  be  classified  using  data  collected  dynamically  using  advanced 
EMI  sensors.  This  raises  the  possibility  of  performing  the  anomaly  detection  step  using  advanced 
sensors,  classifying  many  of  the  anomalies  using  the  same  data  set  (2/3  of  the  anomalies  in  a  recent 
test),  and  only  having  to  collect  cued  data  over  a  subset  of  the  anomalies.  The  cost  estimate  for  this 
“Advanced  Sensor  Only”  mode  is  given  in  the  rightmost  two  columns  in  Table  8-2. 

Given  the  reduced  number  of  data  collections,  one  might  expect  the  savings  from  the  advanced 
sensor  only  mode  to  be  much  greater  than  the  hybrid  approach.  The  current  generation  of 
advanced  sensors  are  not  configured  to  be  deployed  in  arrays  so  the  cost  for  the  advanced  sensor 
detection  survey  at  this  example  site  is  much  higher  than  for  detection  using  an  EM61-MK2  array. 
As  the  operation  of  these  advanced  sensors  in  dynamic  mode  is  explored  further,  it  is  possible  that 
this  cost  disadvantage  will  lessen. 


Table  8-2.  Cost  Comparison  for  100  acres  of  comparable  Spencer  Range  site 


No  Classification 

Hybrid  Approach1 

Advanced  Sensor 
Only2 

Work  Item 

Quantity 

Cost/$ 

Quantity 

Cost/$ 

Quantity 

Cost/$ 

Mob/ demob 

1 

7,000 

1 

7,000 

1 

7,000 

Surface  Sweep 

100  acres 

250,000 

100  acres 

250,000 

100  acres 

250,000 

Seeds 

125  items 

22,650 

125  items 

22,650 

125  items 

22,650 

EM61  Survey 

100  acres 

100,000 

100  acres 

100,000 

n/ a 

MetalMapper 

Survey 

n/ a 

n/ a 

100  acres 

500,000 

Cued 

MetalMapper 

n/ a 

25,000 

anomalies 

750,000 

8,333 

anomalies 

249,990 

Seeds  Dug 

125 

15,625 

125 

15,625 

125 

15,625 

Native  UXO  Dug 

75 

9,375 

75 

9,375 

75 

9,375 

Clutter  Dug 

24,925 

3,115,625 

4,985 

623,125 

4,985 

623,125 

TOTAL  | 

3,520,275 

1,777,775 

1,677,765 

Percent  Savings 

49% 

52% 

1  Hybrid  approach  denotes  the  method  used  in  this  demonstration,  anomaly  detection  from  an  EM61-MK2 
survey  with  cued  data  collection  using  advanced  sensors. 

2  Advanced  sensor  only  approach  refers  to  anomaly  detection  and  classification  of  2/ 3  of  the  anomalies  using 
MetalMapper  in  dynamic  mode  and  cued  data  collection  on  the  remaining  1/3  of  the  anomalies. 

9.  Conclusions 

Production  contractor  field  crews  from  two  different  vendors  collected  high  quality  cued 
MetalMapper  data  used  on  the  former  Spencer  Artillery  Range  Open  Area.  Both  production 
contractor  geophysicists  and  the  developers  of  classification  methods  were  successful  in  using  these 
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data  to  achieve  substantial  classification.  Most  analysts  correctly  identified  all,  or  all  but  one,  TOI 
while  correctly  classifying  75  to  90%  of  the  clutter. 

The  best  predictor  of  classification  performance  was  analyst  experience.  Figure  9-1  repeats  the  left 
panel  of  Figure  7-1  with  the  points  colored  to  denote  the  experience  of  the  analyst.  While  not  all 
novice  analysts  were  at  the  lower  end  of  the  performance  metric  and  not  all  experienced  analysts 
achieved  perfect  classification,  the  results  are  generally  reflective  of  the  analyst’s  experience.  This 
points  to  continued  analyst  training  as  an  important  component  of  technology  transfer  for  these 
technologies. 

As  can  be  seen  in  Figure  9-1,  one  of  the  experienced  analysts  did  not  correctly  classify  one  of  the 
TOI,  an  inert  37-mm  QC  seed.  This  illustrates  the  importance  of  a  well-defined  and  executed  QC 
program. 

The  Spencer  Range  site  is  an  excellent  site  for  classification  using  advanced  EMI  sensors.  The  flat 
terrain  and  corresponding  low  survey  noise  led  to  excellent  detection  and  classification  results. 


Figure  9-1.  Plot  from  the  left  panel  of  Figure  7-1  with  the  points  colored  by  analyst  experience 


10.  Acronyms 

BRAC 

Base  Realignment  and  Closure 

CO 

Colorado 

EM 

Electromagnetic 

EMI 

Electromagnetic  Induction 

ESTCP 

Environmental  Security  Technology  Certification  Program 

FUDS 

Formerly  Used  Defense  Site 

GPS 

Global  Positioning  System 

IMU 

Inertial  Measurement  Unit 

ISO 

Industry  Standard  Object 
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TOI 

UXO 


IVS 

MPV 

QC 


SERDP 


Instrument  Verification  Strip 
Man-portable  Vector  [Sensor] 

Quality  Control 

Strategic  Environmental  Research  and  Development  Program 
Target  of  Interest 
Unexploded  Ordnance 
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